Method and system for tactile-biased sensory-enhanced e-reading

ABSTRACT

A method of synchronizing multi-sensory enhancement with e-reading content is provided. The method includes tracking eye movement of a user of an e-reader, providing a pre-defined multi-sensory indicator embedded within a portion of a story presented on the e-reader and responsive to the eye movement of the user being correlated with the pre-defined multi-sensory indicator, providing a multi-sensory enhancement which is associated with the portion of the story presented on the e-reader.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is related to co-pending U.S. patent application Ser.No. 14/533,890, filed on Nov. 5, 2014, entitled “PROVIDING A SCENT WHILEA USER INTERACTS WITH AN ELECTRONIC MEDIA PROVIDING DEVICE,” by Liu etal., having Attorney Docket No. KOBO-3012, and assigned to the assigneeof the present application and hereby incorporated by reference in itsentirety.

This application is related to co-pending U.S. patent application Ser.No. 14/533,700, filed on Nov. 5, 2014, entitled “OPERATING AN ELECTRONICPERSONAL DISPLAY USING EYE MOVEMENT TRACKING,” by Liu, having AttorneyDocket No. KOBO-3013, and assigned to the assignee of the presentapplication and hereby incorporated by reference in its entirety.

This application is related to co-pending U.S. patent application Ser.No. 14/553,522, filed on Nov. 25, 2014, entitled “AUDIO IN SYNCHRONIZEDOPERATION WITH E-READING CONTENT,” by Flawn et al., having AttorneyDocket No. KOBO-3030, and assigned to the assignee of the presentapplication and hereby incorporated by reference in its entirety.

This application is related to co-pending U.S. patent application Ser.No. ______, filed on _____, entitled “METHOD AND SYSTEM FORVISUALLY-BIASED SENSORY-ENHANCED E-READING,” by Flawn et al., havingAttorney Docket No. KOBO-3038, and assigned to the assignee of thepresent application and hereby incorporated by reference in itsentirety.

This application is related to co-pending U.S. patent application Ser.No. ______, filed on _____, entitled “METHOD AND SYSTEM FOR CUSTOMIZABLEMULTI-LAYERED SENSORY-ENHANCED E-READING INTERFACE,” by Flawn et al.,having Attorney Docket No. KOBO-3042, and assigned to the assignee ofthe present application and hereby incorporated by reference in itsentirety.

TECHNICAL FIELD

Examples described herein relate to a system and method fortactile-biased sensory-enhanced e-Reading.

BACKGROUND

An electronic personal display is a mobile computing device thatdisplays information to a user. While an electronic personal display maybe capable of many of the functions of a personal computer, a user cantypically interact directly with an electronic personal display withoutthe use of a keyboard that is separate from, or coupled to, but distinctfrom the electronic personal display itself. Some examples of electronicpersonal displays include mobile digital devices/tablet computers andelectronic readers (e-readers) such (e.g., Apple iPad®, Microsoft®Surface™ Samsung Galaxy Tab® and the like), handheld multimediasmartphones (e.g., Apple iPhone®, Samsung Galaxy S®, and the like), andhandheld electronic readers (e.g., Amazon Kindle®, Barnes and NobleNook®, Kobo Aura H D, Kobo Aura H2O, Kobo GLO and the like).

Some electronic personal display devices are purpose built devicesdesigned to perform especially well at displaying digitally storedcontent for reading or viewing thereon. For example, a purpose builddevice may include a display that reduces glare, performs well in highlighting conditions, and/or mimics the look of text as presented viaactual discrete pages of paper. While such purpose built devices mayexcel at displaying content for a user to read, they may also performother functions, such as displaying images, emitting audio, recordingaudio, and web surfing, among others.

Electronic personal displays are among numerous kinds of consumerdevices that can receive services and utilize resources across a networkservice. Such devices can operate applications or provide otherfunctionality that links a device to a particular account of a specificservice. For example, the electronic reader (e-reader) devices typicallylink to an online bookstore, and media playback devices often includeapplications that enable the user to access an online media electroniclibrary (or e-library). In this context, the user accounts can enablethe user to receive the full benefit and functionality of the device.

Yet further, such devices may incorporate a touch screen display havingintegrated touch sensors and touch sensing functionality, whereby userinput commands via touch-based gestures are received thereon.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and form a part ofthis specification, illustrate various embodiments and, together withthe Description of Embodiments, serve to explain principles discussedbelow. The drawings referred to in this brief description of thedrawings should not be understood as being drawn to scale unlessspecifically noted.

FIG. 1 illustrates a system utilizing applications and providing e-bookservices on a computing device configured for operation of an e-bookreading launch interface, in an embodiment.

FIG. 2 illustrates a schematic architecture of a computing device forconfiguring and launching an e-book reading interface, according to anembodiment.

FIG. 3 illustrates example embodiments for multi-sensory enhancede-reading.

FIG. 4 illustrates a method for multi-sensory enhanced e-reading,according to an embodiment.

FIG. 5 illustrates an exemplary computer system for providing amulti-sensory enhancement, according to an embodiment.

DETAILED DESCRIPTION

Embodiments include eye tracking while e-reading and providing visualenhancements based on the eye tracking In one embodiment, bookloverswill be able to select an immersive reading experience based onmulti-sensory sensory enhancements. For example, when reaching theclimax of a horror novel (end of chapter or end of book) or whentriggering a specific word such as “murder” or “blood” a faint redlight, or blotches of red light could begin pulsating and may bevibrating behind the text.

Using technology that enables vibration motions, for example, vibrationon mobile phones and tablets, users could choose to “enable touchenhanced mode” and feel vibrations or other motions while reading. Theeye-tracking technology of co-pending U.S. patent application Ser. No.14/533,700, filed on Nov. 5, 2014, entitled “OPERATING AN ELECTRONICPERSONAL DISPLAY USING EYE MOVEMENT TRACKING,” by Liu, having AttorneyDocket No. KOBO-3013, and assigned to the assignee of the presentapplication and hereby incorporated by reference in its entirety,enables certain words to trigger corresponding, immersion-enhancingmotions.

Embodiments described herein provide a multi-sensor enhancement, forexample, when a user reads the word “boat” the device could perform arolling movement (using vibrations) to help the reader feel like they'reon the rolling ocean. When coming across the word “footsteps” the usercould feel alternating vibrations moving up either side of the pagelittle feet walking up their device. The intensity and frequency ofthese word-triggered vibrations could be controlled in the settingsection of the feature or could be turned off completely.

“E-books” are a form of electronic publication content stored in digitalformat in a computer non-transitory memory, viewable on a computingdevice having display functionality. An e-book can correspond to, ormimic, the paginated format of a printed publication for viewing, suchas provided by printed literary works (e.g., novels) and periodicals(e.g., magazines, comic books, journals, etc.). Optionally, some e-booksmay have chapter designations, as well as content that corresponds tographics or images (e.g., such as in the case of magazines or comicbooks). Multi-function devices, such as cellular-telephony or messagingdevices, can utilize specialized applications (e.g., specializede-reading application software) to view e-books in a format that mimicsthe paginated printed publication. Still further, some devices(sometimes labeled as “e-readers”) can display digitally-stored contentin a more reading-centric manner, while also providing, via a user inputinterface, the ability to manipulate that content for viewing, such asvia discrete pages arranged sequentially (that is, pagination)corresponding to an intended or natural reading progression, or flow, ofthe content therein.

An “e-reading device”, variously referred to herein as an electronicpersonal display or mobile computing device, can refer to any computingdevice that can display or otherwise render an e-book. By way ofexample, an e-reading device can include a mobile computing device onwhich an e-reading application can be executed to render content thatincludes e-books (e.g., comic books, magazines, etc.). Such mobilecomputing devices can include, for example, a multi-functional computingdevice for cellular telephony/messaging (e.g., feature phone or smartphone), a tablet computer device, an ultra-mobile computing device, or awearable computing device with a form factor of a wearable accessorydevice (e.g., smart watch or bracelet, glass-wear integrated with acomputing device, etc.). As another example, an e-reading device caninclude an e-reader device, such as a purpose-built device that isoptimized for an e-reading experience (e.g., with E-ink displays).

While conventional physical paper books typically include afixedly-configured table of contents page(s) intended to assist a useror observer to locate a desired portion or page of the book for reading,a digitally rendered e-book may be configured in other, more fluidarrangements that allow alternative ways for a user to convenientlyaccess a particular content portion or page of the e-book.

FIG. 1 illustrates a system 100 for utilizing applications and providinge-book services on a computing device configured for operation of ane-book reading launch interface, according to an embodiment. In anexample of FIG. 1, system 100 includes an electronic personal displaydevice, shown by way of example as an e-reading device 110, and anetwork service 121. The network service 121 can include multipleservers and other computing resources that provide various services inconnection with one or more applications that are installed on thee-reading device 110. For example, in one embodiment, the networkservice 121 may provide multi-sensory enhancements that correspond withe-reading content. By way of example, in one implementation, the networkservice 121 can provide e-book services that communicate with thee-reading device 110. The e-book services provided through networkservice 121 can, for example, include services in which e-books aresold, shared, downloaded and/or stored. More generally, the networkservice 121 can provide various other content services, includingcontent rendering services (e.g., streaming media) or other networkapplication environments or services.

The e-reading device 110 can correspond to any electronic personaldisplay device on which applications and application resources (e.g.,e-books, media files, documents) can be rendered and consumed. Forexample, the e-reading device 110 can correspond to a tablet or atelephony/messaging device (e.g., smart phone). In one implementation,for example, e-reading device 110 can run an e-reader application thatlinks the device to the network service 121 and enables e-books providedthrough the service to be viewed and consumed by way of e-reading. Inanother implementation, the e-reading device 110 can run a mediaplayback or streaming application that receives files or streaming datafrom the network service 121. By way of example, the e-reading device110 can be equipped with hardware and software to optimize certainapplication activities, such as reading electronic content (e.g.,e-books). For example, the e-reading device 110 can have a tablet-likeform factor, although variations are possible. In some cases, thee-reading device 110 can also have an E-ink display.

In additional detail, the network service 121 can include a deviceinterface 128, a content store 122 and a user account electronic library(e-library) 124 storing e-books or digital content items. Content store122 may be an online store for purchasing of digital content items fordownload therefrom onto a resident memory of e-reading device 110 and/oruser account e-library 124. User account e-library 124 associates thee-reading device 110 with a user having an account 123. The account 123can also be associated with ownership of, and/or accessibility to, oneor more digital content items stored in content store 122. In oneembodiment, the digital content items are e-books, and the content store122 is an online store having e-books for purchase or other licenseduse. The device interface 128 can handle requests from the e-readingdevice 110 with regard to services and functionality of the networkservice 121. The device interface 128 can utilize information providedwith user account 123 in order to enable services, such as purchasingand downloading of e-books into user account e-library 124, anddetermining what e-books and content items providable via content store122 are associated with, and accessible to, user account 123.Additionally, the device interface 128 can provide the e-reading device110 with access to the on-line content store 122. The device interface128 can handle input to identify content items (e.g., e-books), andfurther to link content items to the account 123 of the user.

Yet further, the user account e-library 124 can retain metadata forindividual accounts 123 to identify e-books or other digital contentitems that have been purchased or made available for consumption for agiven account. Thus information relating to e-books within user accounte-library 124 can include a metadata set in addition to substantivedigital text and image content portions. The metadata set can include,for example, information such as the graphic representation of thee-book, such as including artwork- or image-based representation of acounterpart physical paper book cover, as well as summary information,author information, title, short synapse or book review, publicationdate and language of the e-book, and book or volume series information.

The e-reading device 110 may be associated with the user account 123,and multiple devices may be associated with the same account. Asdescribed in greater detail below, e-reading device 110 can locallystore content items (e.g., e-books) that are purchased or otherwise madeavailable to the user of the e-reading device 110 as well as to archive,in user account 124, e-books and other digital content items that havebeen purchased for the user account 123, but are not necessarily storedin local resident memory of computing device 110.

With reference to an example of FIG. 1, e-reading device 110 can includea touchscreen display 116. In an embodiment, the display screen 116 istouch-sensitive, to process touch inputs including gestures (e.g.,swipes). For example, the display screen 116 may be integrated with oneor more touch sensors 138 to provide a touch-sensing region on a surfaceof the display screen 116. For some embodiments, the one or more touchsensors 138 may include capacitive sensors that can sense or detect ahuman body's capacitance as input. In the example of FIG. 1, thetouch-sensing region coincides with a substantial surface area, if notall, of the display screen 116.

In some embodiments, the e-reading device 110 includes features forproviding functionality related to displaying paginated content,including paginated content comprising an e-magazine or e-comic book.The e-reading device 110 can include page transitioning logic, whichenables the user to transition through paginated content. The e-readingdevice 110 can display pages of e-books, e-magazines and e-comics, andenable the user to transition from one page state to another. Inparticular, an e-book can provide content that is rendered sequentiallyin pages, and the e-book can display page states in the form of singlepages, multiple pages or portions thereof. Accordingly, a given pagestate can coincide with, for example, a single page, or two or morepages displayed at once. Page transitioning logic can operate to enablethe user to transition from a given page state to another page state Inthe specific example embodiment where a given page state coincides witha single page, for instance, each page state corresponding to one pageof the digitally constructed, ordered sequence of pages paginated tocomprise, in one embodiment, an e-book. In some implementations, thepage transitioning logic enables single page transitions, chaptertransitions, or cluster transitions (multiple pages at one time).

According to some embodiments, the e-reading device 110 includes displaysensor logic 135 to detect and interpret user input or user inputcommands made through interaction with the touch sensors 138. By way ofexample, display sensor logic 135 can detect a user making contact withthe touch-sensing region of the display screen 116, otherwise known as atouch event. More specifically, display sensor logic 135 can detect atouch events also referred to herein as a tap, an initial tap held incontact with display screen 116 for longer than some pre-definedthreshold duration of time (otherwise known as a “long press” or a “longtouch”), multiple taps performed either sequentially or generallysimultaneously, swiping gesture actions made through user interactionwith the touch sensing region of the display screen 116, or anycombination of these gesture actions. Although referred to herein as a“touch” or a tap, it should be appreciated that in some designimplementations, sufficient proximity to the screen surface, evenwithout actual physical contact, may register a “contact” or a “touchevent”. Furthermore, display sensor logic 135 can interpret suchinteractions in a variety of ways. For example, each such interactionmay be interpreted as a particular type of user input associated with arespective input command, execution of which may trigger a change instate of display 116.

In one implementation, display sensor logic 135 implements operations tomonitor for the user contacting or superimposing upon, using a finger,thumb or stylus, a surface of display 116 coinciding with a placement ofone or more touch sensor components 138, that is, a touch event, andalso detects and correlates a particular gesture action (e.g., pinching,swiping, tapping, etc.) as a particular type of input command. Displaysensor logic is also responsive to the user's eye contact with variouswords or text that may initiate presentation of multi-sensoryenhancements that correspond with e-reading content. Display sensorlogic 135 may also sense directionality of a user gesture action so asto distinguish between, for example, leftward, rightward, upward,downward and diagonal swipes along a surface portion of display screen116 for the purpose of associating respective user input commandstherewith.

E-library view (or interface) logic 120 provides an interface,displayable via display screen 116 of computing device 110, showingtitles in a user's e-library collection of e-books, or from a user'shome page in relation to an online content store 122 hosting e-books forcommercial sale and downloading therefrom. The e-library collection ofe-books may be hosted via a remotely located computer server deviceassociate with user account e-library 124, or at a locally residentwithin a memory at computing device 110. The e-library view logic 120can display iconic or other graphic representations of individuale-books in the user's e-library collection. For example, the e-libraryview logic 120 can use the metadata associated with the records of thee-books in the user's e-library account 124 to display lists, folders,or other virtual structures that include graphic representations and/orother identifiers of e-books in the user's collection. The metadata setcan include, for example, information such as the graphic representationof the e-book, such as including artwork- or image-based representationof a counterpart physical paper book cover, as well as summaryinformation, author information, title, short synapse or book review,publication date and language of the e-book, and book or volume seriesinformation. The user's collection can include e-books that the user hason the particular device 110 (e.g., locally stored e-books), as well ase-books that are not locally stored, but rather are stored or archivedat a remote computer server and associated with the user accounte-library 124.

Annotations interface logic module 125 provides an annotations andbookmarking scheme in conjunction with the interface rendered viae-library view logic 120, providing an annotations interface page(s) tobe deployed upon launch in lieu of a table of contents or a first pageof an e-book for reading. Launch of the e-book for reading, in oneembodiment, is triggered by a user enacting a touch event upon agraphical icon representing a specific e-book from an e-librarycollection, as will be described further in regard to FIGS. 2 and 3.

E-library view logic module 120 and annotations interface logic module125 can be implemented as software modules comprising instructionsstored in a memory of mobile computing device 110, as described infurther detail below with regard to FIG. 2.

In one or more embodiments of e-library view logic module 120, displaysensor logic 135 and annotations interface logic module 125 describedherein may be implemented using programmatic modules or components. Aprogrammatic module or component may include a program, a subroutine, aportion of a program, or a software or a hardware component capable ofperforming one or more stated tasks or functions in conjunction with oneor more processors. As used herein, a module or component can exist on ahardware component independently of other modules or components.Alternatively, a module or component can be a shared element or processof other modules, programs and hardware components.

Furthermore, the one or more embodiments of e-library view logic module120, display sensor logic 135 and annotations interface logic module 125described herein may be implemented through instructions that areexecutable by one or more processors. These instructions may be storedon a computer-readable non-transitory medium. In particular, thenumerous computing and communication devices shown with embodiments ofthe invention include processor(s) and various forms of computer memory,including volatile and non-volatile forms, storing data andinstructions. Examples of computer-readable mediums include permanentmemory storage devices, such as hard drives on personal computers orservers. Other examples of computer storage mediums include portablestorage units, flash or solid-state memory (such as included on manycell phones and consumer electronic devices) and magnetic memory.Computers, terminals, network enabled devices (e.g., mobile devices suchas cell phones and wearable computers) are all examples of machines anddevices that utilize processors, memory, and instructions stored oncomputer-readable mediums. Additionally, embodiments may be implementedin the form of computer-programs, or a computer usable storage mediumcapable of storing such a program.

FIG. 2 illustrates a schematic architecture of a computing device forconfiguring and launching an e-book reading interface, according to anembodiment.

E-reading device 110 further includes processor 210, a memory 250storing instructions and logic pertaining at least to display sensorlogic 135, e-library view logic module 120 and annotations interfacelogic 125.

Processor 210 can implement functionality using the logic andinstructions stored in memory 250. Additionally, in someimplementations, processor 210 communicates with the network service 121(see FIG. 1). More specifically, the e-reading device 110 can access thenetwork service 121 to receive various kinds of resources (e.g., digitalcontent items such as e-books, configuration files, accountinformation), as well as to provide information (e.g., user accountinformation, service requests etc.). For example, e-reading device 110can receive application resources, such as e-books or media files, thatthe user elects to purchase or otherwise download via the networkservice 121. The application resources that are downloaded onto thee-reading device 110 can be stored in memory 250.

In some implementations, display 116 can correspond to, for example, aliquid crystal display (LCD) or light emitting diode (LED) display thatilluminates in order to provide content generated from processor 210. Insome implementations, display 116 can be touch-sensitive. For example,in some embodiments, one or more of the touch sensor components 138 maybe integrated with display 116. In other embodiments, the touch sensorcomponents 138 may be provided (e.g., as a layer) above or below display116 such that individual touch sensor components 138 track differentregions of display 116. Further, in some variations, display 116 cancorrespond to an electronic paper type display, which mimicsconventional paper in the manner in which content is displayed. Examplesof such display technologies include electrophoretic displays,electro-wetting displays, and electro-fluidic displays.

Processor 210 can receive input from various sources, including touchsensor components 138, display 116, keystroke input 208 such as from avirtual or rendered keyboard, and other input mechanisms 299 (e.g.,buttons, mouse, microphone, etc.). With reference to examples describedherein, processor 210 can respond to input detected at the touch sensorcomponents 138. In some embodiments, processor 210 responds to inputsfrom the touch sensor components 138 in order to facilitate or enhancee-book activities such as generating e-book content on display 116,performing page transitions of the displayed e-book content, poweringoff the device 110 and/or display 116, activating a screen saver,launching or closing an application, and/or otherwise altering a stateof display 116.

In some embodiments, memory 250 may store display sensor logic 135 thatmonitors for user interactions detected through the touch sensorcomponents 138, and further processes the user interactions as aparticular input or type of input. In an alternative embodiment, displaysensor logic module 135 may be integrated with the touch sensorcomponents 138. For example, the touch sensor components 138 can beprovided as a modular component that includes integrated circuits orother hardware logic, and such resources can provide some or all ofdisplay sensor logic 135. In variations, some or all of display sensorlogic 135 may be implemented with processor 210 (which utilizesinstructions stored in memory 250), or with an alternative processingresource.

E-reading device 110 further includes wireless connectivity subsystem213, comprising a wireless communication receiver, a transmitter, andassociated components, such as one or more embedded or internal antennaelements, local oscillators, and a processing module such as a digitalsignal processor (DSP) (not shown). As will be apparent to those skilledin the field of communications, the particular design of wirelessconnectivity subsystem 213 depends on the communication network in whichcomputing device 110 is intended to operate, such as in accordance withWi-Fi, Bluetooth, Near Field Communication (NFC) communicationprotocols, and the like.

E-library view logic module 120 can be implemented as a software module,comprising instructions stored in memory 250, on mobile computing device110. In one implementation, the local memory 250 can include records foreach e-book in the user's e-library account 124, each record includemetadata of the e-books therein. The user may have the content portionof select e-books archived remotely at a computer server cloud system,so as not to reside in the local memory 250, but be provided by thenetwork service 121 upon request or as needed. By way of example, thee-library view logic module 120 can display the e-books of a user'scollection in the form of a virtual bookshelf or bookcase featureshowing graphical icons representing the e-books. In such animplementation, the e-books are displayed as icons that include imagery,title information, etc. In a variation, the e-library view module 120can display representations of e-books in the user's collection asicons, or as icons with associated text. Still further, folders can beused to provide a panel view of the graphic representations (e.g., iconsand/or text) of the e-books in the user's e-library collection 124,corresponding to a side view of a bookshelf showing book spines withtitles printed thereon for identifying individual books.

Annotations interface logic 125 can be implemented as a software modulecomprising instructions stored in memory 250 of computing device 110Annotations interface logic module 125 provides a provides anannotations and bookmarking interface scheme in conjunction withe-library view logic 120, configuring an annotations interface page(s),which can be deployed upon a subsequent launch of an e-book for reading.In embodiment, upon e-book launch for reading, the annotations interfacepage can be presented in lieu of a typical table of contents or a firstsubstantive reading page. Launch of the e-book for reading may betriggered by a user enacting a touch event upon a graphical iconrepresenting a specific e-book from e-library collection 124 asdisplayed on display screen 116 via e-library view logic 120.

FIG. 3 illustrates embodiments of providing multi-sensory enhancede-reading. Embodiments include eye tracking while e-reading andproviding multi-sensory enhancements based on the eye tracking In oneembodiment, sight is used to enhance the e-reading experience of a userand in one embodiment, multi-sensory enhancements are provided to theuser that are related to particular pre-defined multi-sensory indicatoror trigger words. For example, triggers include words such as“footsteps” 350 or phrases on the page the user is reading. In the caseof the word “footsteps” 350, vibrating footsteps 321, 322, 323 and 324may be displayed on the device and may move lift to right up the screenlike alternating footsteps after the user reads the trigger wordfootsteps 350 to give the user a sensation that someone is running upthe screen.

The multi-sensory enhancements may be specific to a particular story,genre, or e-reading setting. In one embodiment, booklovers will be ableto select an immersive reading experience based on multi-sensoryenhancements. In the case of the vibrating footsteps, the userexperiences a touch and visual sensation that more fully immerses theuser in the story. In another example, when reaching the climax of ahorror novel (end of chapter or end of book) or when triggering aspecific word such as “murder” or “blood” a faint red light, or blotchesof red light could begin pulsating behind the text. In anotherembodiment, when a user is reading a thriller or mystery book, a bullethole may appear as if a bullet had been shot through the e-reader.

It is appreciated that the multi-sensory enhancements may be accesses asa stored file and may be accessed from a remote location.

In one embodiment, the multi-sensory enhancement could appear in thebackground, in the margins of the page, or translucency. In a book aboutthe sea, the just-read word “ocean” could trigger blue illumination inthe background or subtle ripples or vibrations behind the text likewaves on the surface of the sea. Embodiments include a multi-layeredsensory-driven reading experience for sight that includes an extensiveelectronic depository of words that trigger corresponding images orother visual enhancements such as the examples above. The feature couldalso be customizable, allowing users to program certain words to triggerparticular images or image types.

Next with reference to FIG. 4, illustrated is a method 400 for providingmulti-sensory enhancement to an E-reading experience, according to anembodiment. In describing the example of FIG. 4, reference will be madeto components such as described with regard to FIGS. 1 through 3 forpurposes of illustrating components for performing a step or sub-step asdescribed.

At step 402, method 400 includes tracking eye movement of a user of ane-reader. Co-pending U.S. patent application Ser. No. 14/533,700, filedon Nov. 5, 2014, entitled “OPERATING AN ELECTRONIC PERSONAL DISPLAYUSING EYE MOVEMENT TRACKING,” by Liu, having Attorney Docket No.KOBO-3013, and assigned to the assignee of the present application andhereby incorporated by reference in its entirety provides details fortracking eye movement according to embodiments described herein.

At step 404, method 400 includes providing a pre-defined multi-sensoryindicator to trigger embedded with a portion of a story presented on thee-reader. In one embodiment, a library containing multi-sensoryenhancements and corresponding trigger or indicator words is accessedwhen e-book content is loaded and when a user views particular triggeror indicator words or phrases, corresponding multi-sensory enhancementsfrom the library can be accessed and presented to the user.

At 406, method 400 includes responsive to the eye movement of the userbeing correlated with the pre-defined multi-sensory indicator, providinga multi-sensory enhancement which is associated with the portion of thestory presented on the e-reader. In one embodiment, the predefinedmulti-sensor indicator is a word or phrase on the page that is displayedon the e-reading device.

Example Computer System Environment

With reference now to FIG. 5, all or portions of some embodimentsdescribed herein are composed of computer-readable andcomputer-executable instructions that reside, for example, incomputer-usable/computer-readable storage media of a computer system.That is, FIG. 5 illustrates one example of a type of computer (computersystem 500) that can be used in accordance with or to implement variousembodiments of an e-Reader, such as e-Reader 110, which are discussedherein. It is appreciated that computer system 500 of FIG. 5 is only anexample and that embodiments as described herein can operate on orwithin a number of different computer systems.

System 500 of FIG. 5 includes an address/data bus 504 for communicatinginformation, and a processor 210A coupled to bus 504 for processinginformation and instructions. As depicted in FIG. 5, system 500 is alsowell suited to a multi-processor environment in which a plurality ofprocessors 210A, 210B, and 210C are present. Processors 210A, 210B, and210C may be any of various types of microprocessors. For example, insome multi-processor embodiments, one of the multiple processors may bea touch sensing processor and/or one of the processors may be a displayprocessor. Conversely, system 500 is also well suited to having a singleprocessor such as, for example, processor 210A.

System 500 also includes data storage features such as a computer usablevolatile memory 508, e.g., random access memory (RAM), coupled to bus504 for storing information and instructions for processors 210A, 210B,and 210C. System 500 also includes computer usable non-volatile memory510, e.g., read only memory (ROM), coupled to bus 504 for storing staticinformation and instructions for processors 210A, 210B, and 210C. Alsopresent in system 500 is a data storage unit 512 (e.g., a magnetic oroptical disk and disk drive) coupled to bus 504 for storing informationand instructions.

Computer system 500 of FIG. 5 is well adapted to having peripheralcomputer-readable storage media 502 such as, for example, a floppy disk,a compact disc, digital versatile disc, universal serial bus “flash”drive, removable memory card, and the like coupled thereto. In someembodiments, computer-readable storage media 502 may be coupled withcomputer system 500 (e.g., to bus 504) by insertion into removable astorage media slot.

System 500 also includes or couples with display 116 for visiblydisplaying information such as alphanumeric text and graphic images. Insome embodiments, system 500 also includes or couples with one or moretouch sensors 138 for communicating information, cursor control, gestureinput, command selection, and/or other user input to processor 210A orone or more of the processors in a multi-processor embodiment. In someembodiments, system 500 also includes or couples with one or moreoptional speakers 150 for emitting audio output. In some embodiments,system 500 also includes or couples with an optional microphone 160 forreceiving/capturing audio inputs. In some embodiments, system 500 alsoincludes or couples with an optional digital camera 170 forreceiving/capturing digital images as an input.

Optional touch sensor(s) 138 allows a user of computer system 500 (e.g.,a user of an eReader of which computer system 500 is a part) todynamically signal the movement of a visible symbol (cursor) on display116 and indicate user selections of selectable items displayed. In someembodiment other implementations of a cursor control device and/or userinput device may also be included to provide input to computer system500, a variety of these are well known and include: trackballs, keypads,directional keys, and the like.

System 500 is also well suited to having a cursor directed or user inputreceived by other means such as, for example, voice commands receivedvia microphone 160. System 500 also includes input/output (I/O) devices520 for coupling system 500 with external entities. For example, in oneembodiment, I/O device 520 is a modem for enabling wired communicationsor modem and radio for enabling wireless communications between system500 and an external device and/or external network such as, but notlimited to, the Internet. I/O device 520 may include a short-rangewireless radio such as a Bluetooth® radio, Wi-Fi radio (e.g., a radiocompliant with Institute of Electrical and Electronics Engineers' (IEEE)802.11 standards), or the like.

Referring still to FIG. 5, various other components are depicted forsystem 500. Specifically, when present, an operating system 522,applications 524, modules 526, and/or data 628 are shown as typicallyresiding in one or some combination of computer usable volatile memory508 (e.g., RAM), computer usable non-volatile memory 510 (e.g., ROM),and data storage unit 512. For example, modules 526 may include anapplication module for providing a page flipping mode for a user.

In some embodiments, all or portions of various embodiments describedherein are stored, for example, as an application 524 and/or module 526in memory locations within RAM 508, ROM 510, computer-readable storagemedia within data storage unit 512, peripheral computer-readable storagemedia 502, and/or other tangible computer readable storage media.

Although illustrative embodiments have been described in detail hereinwith reference to the accompanying drawings, variations to specificembodiments and details are contemplated and encompassed by thisdisclosure. It is intended that the scope of embodiments describedherein be defined by claims and their equivalents. Furthermore, it iscontemplated that a particular feature described, either individually oras part of an embodiment, can be combined with other individuallydescribed features, or parts of other embodiments. Thus, absence ofdescribing combinations should not preclude the inventor(s) fromclaiming rights to such combinations.

What is claimed is:
 1. A method of synchronizing multi-sensoryenhancement with e-reading content, the method comprising: tracking eyemovement of a user of an e-reader; providing a pre-defined multi-sensoryindicator embedded within a portion of a story presented on thee-reader; and responsive to the eye movement of the user beingcorrelated with the pre-defined multi-sensory indicator, providing amulti-sensory enhancement which is associated with the portion of thestory presented on the e-reader.
 2. The method as recited by claim 1,further comprising: providing a pre-defined background multi-sensoryenhancement indicator embedded within a portion of a story presented onthe e-reader; and responsive to the eye movement of the user beingcorrelated with the pre-defined background multi-sensory enhancementindicator, accessing a multi-sensory enhancement file which isassociated with the portion of the story presented on the e-reader. 3.The method as recited by claim 1, further comprising: providing apre-defined momentary multi-sensory enhancement indicator embeddedwithin a portion of the story presented on the e-reader; and responsiveto the eye movement of the user being correlated with the pre-definedmomentary multi-sensory enhancement indicator, accessing a multi-sensoryenhancement file associated with the portion of the story presented onthe e-reader.
 4. The method as recited by claim 1, further comprising:tracking the eye movement of the user at a line-by-line granularity. 5.The method as recited by claim 1, further comprising: tracking the eyemovement of the user at a word-by-word granularity.
 6. The method asrecited by claim 1, further comprising: providing a pre-defined newsetting indicator embedded within a portion of a story presented on thee-reader; and responsive to the eye movement of the user beingcorrelated with the pre-defined new setting indicator, removing saidmulti-sensory enhancement.
 7. The method as recited by claim 1, furthercomprising: providing a pre-defined fade out indicator embedded within aportion of a story presented on the e-reader; and responsive to the eyemovement of the user being correlated with the pre-defined fade outindicator, fading out the multi-sensory enhancement.
 8. A system thatsynchronizes multi-sensory enhancement with e-reader content on ane-reader, the system comprising: a camera that tracks an eye movement ofa user of the e-reader; a gaze to pre-defined multi-sensory enhancementindicator region correlation logic correlates a gaze of the user with amulti-sensory enhancement file embedded within a portion of a storypresented on the e-reader; and an operation implementation responsive togaze logic implements presentation of the multi-sensory enhancement filein response to the gaze being correlated with the pre-defined visualenhancement indicator region.
 9. The system of claim 8, wherein themulti-sensory enhancement file embedded within the portion of the storyis an animation.
 10. The system of claim 8, wherein the multi-sensoryenhancement file embedded within the portion of the story comprises apicture and a vibration command.
 11. The system of claim 8, wherein thecamera tracks the eye movement of the user of the e-reader at aline-by-line granularity.
 12. The system of claim 8, wherein the cameratracks the eye movement of the user of the e-reader at a word-by-wordgranularity.
 13. The system of claim 8, wherein the gaze to pre-definedmulti-sensory enhancement indicator region correlation logic correlatesthe gaze of the user with a new setting indicator region embedded withinanother portion of the story presented on the e-reader; and theoperation implementation responsive to gaze logic ceases thepresentation of the multi-sensory enhancement file in response to thegaze being correlated with the new setting indicator region.
 14. Thesystem of claim 8, wherein the gaze to pre-defined multi-sensoryenhancement indicator region correlation logic correlates the gaze ofthe user with a pre-defined fade out indicator region embedded withinanother portion of the story presented on the e-reader; and theoperation implementation responsive to gaze logic fades out thepresentation of the multi-sensory enhancement file in response to thegaze being correlated with the pre-defined fade out indicator region.15. The system of claim 8, wherein the multi-sensory enhancement fileembedded within a portion of a story presented on the e-reader is astand-alone add on file for a pre-existing e-book file.
 16. Anon-transitory computer-readable storage medium storing instructionsthat, when executed by a hardware processor of a computing device, causethe hardware processor to perform a method of synchronizingmulti-sensory enhancement with e-reading content, the method comprising:tracking eye movement of a user of an e-reader with a camera of thee-reader; providing a pre-defined multi-sensory enhancement indicatorembedded within a portion of the story presented on the e-reader; andresponsive to the eye movement of the user being correlated with thepre-defined multi-sensory enhancement indicator, presenting amulti-sensory enhancement file associated with the portion of the storypresented on the e-reader.
 17. The non-transitory computer-readablestorage medium as recited by claim 16, further comprising: providing apre-defined new setting indicator embedded within a portion of a storypresented on the e-reader; and responsive to the eye movement of theuser being correlated with the pre-defined new setting indicator,ceasing presentation of the multi-sensory enhancement file.
 18. Thenon-transitory computer-readable storage medium as recited by claim 16,further comprising: providing a pre-defined fade out indicator embeddedwithin a portion of a story presented on the e-reader; and responsive tothe eye movement of the user being correlated with the pre-defined fadeout indicator, fading out the presentation of the multi-sensoryenhancement file.
 19. The non-transitory computer-readable storagemedium as recited by claim 16, further comprising: tracking the eyemovement of the user at a line-by-line granularity.
 20. Thenon-transitory computer-readable storage medium as recited by claim 16,further comprising: tracking the eye movement of the user at aword-by-word granularity.